Compacting regression-suites on-the-fly

Testing is one of the biggest problems of the software industry. Coverage is the main technique for showing that the testing has been thorough. Coverage can be used to find a good regression-suite, i.e. a set of tests that is run on the application after software or data changes in order to check that no new bugs were introduced. This paper is about the experience gained in IBM Haifa Research Lab (HRL) in creating regression-suites and minimizing their size, while maintaining high quality as measured in coverage. The problem we solve, while similar to the one addressed in the literature, has a key difference; the compaction algorithm is implemented on-line due to the large number of tests processed. We compare strategies for implementing on-line set-cover. The trade-offs are between quality of solution (as expressed by the size of the cover), the size of the intermediate sets and computational resources. We show that it is possible to start discarding tests very early without getting a significantly larger final set.